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The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH{S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above Is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 1 33). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )1E1 Responsive to communication(s) filed on 6/16/2003 . 
2a)n This action is FINAL. 2b)IEI This action is non-final. 

3) n Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 
Disposition of Claims 

4) ^ Claim(s) 1-6 and 10-23 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) 0 Claim(s) is/are allowed. 

6) 13 Claim(s) 1-6 and 10-23 is/are rejected. 
?)□ Claim(s) is/are- objected to. 

8) 0 Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) 13 The specification is objected to by the Examiner 

10) 3 The drawing(s) filed on 3/08/2000 is/are: a)n accepted or b)3 objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 

11) 0 The proposed drawing correction filed on is: a)C] approved b)n disapproved by the Examiner. 

If approved, corrected drawings are required In reply to this Office action. 

12) 0 The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§119 and 120 

13) n Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 

a)nAII b)n Some*c)n None of: 

1 .□ Certified copies of the priority documents have been received. 

2. n Certified copies of the priority documents have been received in Application No. . 

3. n Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

14) 3 Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 119(e) (to a provisional application). 

a) □ The translation of the foreign language provisional application has been received. 

15) 0 Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121. 
Attachment(s) 

1 ) 3 Notice of References Cited (PTO-892) 4) Q Interview Summary (PTO-41 3) Paper No(s). . 



2) O Notice of Draftsperscn's Patent Drawing Review (PTO-948) 5) □ Notice of Informal Patent Application (PTO-1 52) 

3) S Information Disclosure Statement(s) (PTO-1449) Paper No(s) 3 . 6) O Other: 



U.S. Patent and Trademark Office 
PTO-326 (Rev. 04-01) 
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DETAILED ACTION 
Election/Restrictions 

1 . The cancellation of claims 7-9 and 24-27 overcomes the examiners restriction 
requirement cited in paragraph 1 of the previous office action. The applicant elected species I to 
examine on the merits without traverse in paper no. 6, filed on June 16, 2003. 

Drawings 

2. The drawings are objected to under 37 CFR 1 .83(a). The drawings must show every 
feature of the invention specified in the claims. Therefore, the method of claims 1-6 and 10-23 
must be shovm or the feature(s) canceled from the claim(s). No new matter should be entered. 

A proposed drawing correction or corrected drawings are required in reply to the Office 
action to avoid abandonment of the application. The objection to the drawings will not be held 
in abeyance. 

Information Disclosure Statement 

3. The listing of references in the specification on pages 2-3 is not a proper information 
disclosure statement. 37 CFR 1.98(b) requires a list of all patents, publications, or other 
information submitted for consideration by the Office, and MPEP § 609 A(l) states, "the list may 
not be incorporated into the specification but must be submitted in a separate paper." Therefore, 
unless the references have been cited by the examiner on form PTO-892, they have not been 
considered. 

Claim Rejections - 35 USC §102 

4. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 
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A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public use or on 
sale in this country, more than one year prior to the date of application for patent in the United States. 

5. Claims 1-3, 6, 1 1, and 16 are rejected under 35 U.S.C. 102(b) as being anticipated by 
Sabata et al (B. Sabata and J. Aggarwal. Estimation of motion from a pair of range images: A 
review. CVGIP, 54(3): 309-324, Nov. 1991.). 
Referring to claim 1 , 

a. Obtaining dense range data that describes the distance of points on the figure from 
a reference is explained by Sabata et al on page 310, second column, third paragraph. 
Sabata et al explain that the range image is the 3D coordinate of each point (or pixel) 
sensed. A 3D coordinate inherently includes both the horizontal and vertical coordinates 
of the object in space and also the distance of the object from the image sensor. 

b. Processing the dense range data to estimate the pose of the figure is explained by 
Sabata et al on page 310, second column, third paragraph. Sabata et al explain that the 
motion of a rigid object in an image is estimated, corresponding to the pose of the object 
("pose determination" on page 309, first column, second paragraph). 

Referring to claim 2, the dense range data being processed in accordance with a set of 
depth constraints to estimate the pose is explained by Sabata et al on page 312, first column, first 
and second paragraphs. Sabata et al explain on page 311, first column, forth paragraph that the 
pose estimation is determined by both a rotational component (R) and a translational component 
(T). Sabata et al then provide constraints to simplify the calculation of the rotation matrix to 
estimate the rotational component of the pose estimation. 
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Referring to claim 3, the depth constraints being linear is explained by Sabata et al on 
page 311, first column, forth paragraph. Sabata et al explain that the rotational component of the 
pose estimation, which is constrained corresponding to claim 2, is a linear transform. 

Referring to claim 6, the dense range data being compared with an estimate of pose to 
produce an error value and iteratively revising the estimate to minimize the error is explained by 
Sabata et al on page 311, first column, second and third paragraph. Sabata et al explain 
determining the error (Best) of the motion estimations and minimizing the error. 

Referring to claim 1 1, 

a. Obtaining dense range data that describes the distance of points on the figure from 
a reference corresponds to claim la. 

b. Processing the dense range data in accordance with a set of linear depth 
constraints to estimate the pose of the figure corresponds to claim 2. 
Referring to claim 16, 

a. The estimate of the pose of the object including an estimate for each of the 
orientation and translational positions of the object is explained by Sabata et al on page 
310, first colunm, forth paragraph. Sabata et al explain estimating the rotational 
component, corresponding to the orientation, and the translational component of the 
motion of an object. 

b. Decoupling the estimate of orientation from the estimate of the translational 
position is explained by Sabata et al on page 311, second column, first paragraph. Sabata 
et al clearly illustrate that the position of point on a succeeding frame is represented by 
the rotational component and the translational component independently. 
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6. Claim 1 is rejected under 35 U.S.C. 102(b) as being anticipated by Kanade et al (U.S. 
Patent No. 6,084,979). 

Referring to claim 1, 

a. Obtaining dense range data that describes the distance of points on the figure from 
a reference is explained by Kanade et al in column 7, lines 1 1-20. Kanade et al explain 
that dense maps are used in motion estimation because of their usefulness in image 
reconstruction. 

b. Processing the dense range data to estimate the pose of the figure is illustrated by 
Kanade et al in figures 15a to 151, wherein the pose of the figure in multiple successive 
frames is estimated. 

Claim Rejections - 35 USC § 103 

7. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

8. Claims 4-5, 10, 12-15, 17, and 21-23 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Sabata et al in view^ of Bregler et al (Bregler, C; Malik, J. Tracking people 
with tmsts and exponential maps. 1998 IEEE Computer Society Conference on Computer 
Vision and Pattern Recognition, 23-25 June 1998. Page(s): 8 -15.). 

Referring to claims 4, 10, and 15, obtaining brightness data in accordance with a set of 
linear brightness constraints to estimate the pose is not expHcitly explained by Sabata et al. 
Sabata et al do explain that the intensity (corresponding to the brightness) of an image is widely 
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used in the art to determine the motion of an object on page 309, first column, second paragraph 
to page 309, second column, first paragraph. However, Sabata et al do not explain linear 
brightness constraints on the intensity data. Bregler et al do explain linear constraints on the 
brightness data on page 9, first column, forth through seventh paragraphs. Bregler et al explain 
constructing an over-constrained set of equations for the motion model, assuming that the motion 
is linear. By applying the linearly constrained brightness data into the motion data determined 
by the range data of Sabata et al, the motion of an object through a series of firames will be 
estimated more accurately. Therefore, it would have been obvious to one of ordinary skill in the 
art at the time the invention was made to use brightness data in accordance with a set of linear 
brightness constraints to better determine the pose of an object. 

Referring to claims 5 and 1 3, the depth constraints being represented by twist 
mathematics is not explicitly explained by Sabata et al. However, Bregler et al do explain using 
twist mathematics in order to estimate the pose of a figure on page 8, second colunm, first 
paragraph. Bregler et al explain that using twdst mathematics in motion estimation, such as the 
estimation of Sabata et al, greatly simplifies the estimation and leads to a more robust tracking 
result on page 8, second column, second paragraph. Therefore, it would have been obvious to 
one of ordinary skill in the art at the time the invention was made to use twist mathematics as 
suggested by Bregler et al in the motion estimation system of Sabata et al in order to perform the 
motion estimation and object tracking more robustly. 

Referring to claim 12, the object being articulated is not explicitly explained by Sabata et 
al. However, Bregler et al do illustrate estimating the pose of an articulated object in figure 2. 
The system of Sabata et al models the motion of an object by the rotational and translation 
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components of the object's movement. Thus, an articulated object could be efficiently tracked in 
the system of Sabata et al. Therefore, it would have been obvious to one of ordinary skill in the 
art at the time the invention was made to track an articulated object using the system of Sabata et 
al. 

Referring claim 14, 

a. Mapping the parameters that describe the rotation and translation of the object to 
a set of linear parameters is explained by Sabata et al on page 311, second column, 
second paragraph. Sabata et al explain that the rotation and translation components of the 
object movement are characterized by individual matrices. 

b. Solving for the depth constraints is explained by Sabata et al on page 312, first 
column, first and second paragraph. The rotational depth constraints are explained by 
Sabata et al and used to estimate the object motion. 

c. Re-mapping back to the original parameters to provide a pose estimate is 
explained by Sabata et al on page 313, second column, fifth paragraph to page 314, first 
colunm, first paragraph. Sabata et al explain that the parameters which determine the 
rotational and translational components of the object are used in order to solve for the 
motion of the object. 

Referring to claim 17, the reference comprising a location on the object and the pose 
being estimated from the positions of points on the object relative to the locations is explained by 
Sabata et al on page 310, second column, sixth paragraph to page 311, first column, second 
paragraph. Sabata et al explain determining the current position of an object (P') from the 
previous position of the object (P) after the determined transformation of the object. 
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Referring to claim 21, 

a. Obtaining dense brightness data for pixels in each of the video images is 
explained by Sabata et al on page 309, first column, second paragraph. 

b. Obtaining dense range data for pixels in each of the video images corresponds to 
claim la. 

c. Determining an initial pose for the object in one of the video images is explained 
by Sabata et al on page 311, first column, forth paragraph by p, which is the initial point 
on an object undergoing a transformation. 

d. Estimating changes in at least one of the translational position and rotational 
orientation of the object for successive images on the basis of the brightness data and 
range data to estimate the pose of the object in successive images corresponds to claims 
lb and 4, wherein the pose is determined using the range data and the brightness data, 
respectively. 

Referring to claim 22, the object being articulated corresponds to claim 12. 

Referring to claim 23, the estimates being obtained by means of linear constraint 
equations that are applied to the brightness data and the range data corresponds to claims 4 and 3, 
respectively. 

9. Claims 18-20 are rejected under 35 U.S.C. 103(a) as being unpatentable over Sabata et al 
in view of Dubuisson- Jolly et al (U.S. Patent No. 6,195,445). 

Referring to claim 1 8, the pose of an object being estimated for each image in a sequence 
of images and selecting a rigid translation value for each point on the object from one image to 
the next is not explicitly explained by Sabata et al. However, Dubuisson- Jolly et al do illustrate 
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rigid translation values for an articulated object in figure 4 by segment lengths Oi toO? and 
explain polyline contours in column 6, lines 49-65. Dubuisson-Jolly et al explain that the 
polyline contours are tracked in an image sequence using an initial polyline contour in an input 
frame in column 6, lines 66 to column 7, line 10. By using the rigid translation value method of 
Dubuisson-Jolly et al, the processing required to track the motion of an articulated object would 
be reduced in the system of Sabata et al. Therefore, it would have been obvious to one of 
ordinary skill in the art at the time the invention was made to use a rigid translation value for 
each segment of an articulated object in order to track the motion of an object more efficiently. 

Referring to claim 19, the rigid translation value being an integer value corresponds to 
claim 1 8, wherein cDj to cD? may represent integer values of the rigid translation. 

Referring to claim 20, the rigid translation values being different for different points on 
the object corresponds to clam 18, wherein each segment, D, has a specific and different rigid 
translation value O. 

Conclusion 

1 0. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. 

Aggarwal et al (Aggarwal, J.K.; Cai, Q.; Human Motion Analysis: A Review. 
Nonrigid and Articulated Motion Workshop, 1997. Proceedings., IEEE , 16 June 
1997 Page(s): 90 -102) - To exhibit tracking an articulated object using multiple 
techniques. 

Sasada et al (U.S. Patent No. 5,692,061) - To using a distance image to determine 
the position and posture of an object as illustrated in figure 1. 
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Darrell et al (U.S. Patent No. 6,445,810) - To exhibit range computation and 



object detection as illustrated in figures 1 and 2. 



Gibas (U.S. Patent No. 5,675,377) - To exhibit using a range image and intensity 



image to determine a scene as illustrated in figures 4, 6, 19B, and 20. 



1 1 . Any inquiry concerning this commimication or earlier communications from the 
examiner should be directed to Hussein Akhavannik whose telephone number is (703)306-4049. 
The examiner can normally be reached on M-F 8:30-5:00. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Leo H. Boudreau can be reached on (703)305-4706. The fax phone numbers for the 
organization where this application or proceeding is assigned are (703)872-9314 for regular 
communications and (703)872-9314 for After Final communications. 

Any inquiry of a general nature or relating to the status of this application or proceeding 
should be directed to the receptionist whose telephone number is (703)305-3900. 



July 11,2003 



Hussein Akhavannik 





